1. Field of the Invention
The present invention relates to methods of processing signal samples representative of a colour video image to produce a legalised colour version of the image. Furthermore, the present invention relates to apparatuses for processing signal samples representative of a colour video image to produce a legalised colour version of the image.
2. Description of the Prior Art
It is well known that the colours of the rainbow, which correspond to light with a range of wavelengths which is visible to the human eye, can be represented from combinations of the colours red, green and blue. For this reason colour television and video images are generated by separating the red, green and blue components of the images and sampling these components at spatially separated sampling points within the image. For example, colour television cameras are provided with a dichronic element which separates the colours of an image formed within a field of view of the camera into red, green and blue components. Each of the red, green and blue components of the image is sampled in two dimensions in accordance with a row-by-column de-composition of the image. Each row is sampled at regularly displaced sampling points to produce a number of samples representing the row which produces the row-by-column de-composition of the image. These sampling points are known to those skilled in the art as pixels. Each of the samples represents one of the red, green and blue components of one of the pixels which make up the image.
The colour image may be re-generated from the signal samples using a colour visual display unit, by separating the signal samples representing the red, green and blue components of the pixels and feeding each respectively to one of three image generators. Each of the image generators operates to reconstruct, row-by-column, a version of the image for one of the three colours of red, green or blue which are super-imposed on a colour screen. By producing the red, green and blue components of each pixel at positions on the screen corresponding to the positions of the pixels from which the colour image was sampled, the colour image is re-generated. Since each pixel is comprised of red, green and blue components, the relative intensity of these components produces a mixture of red, green and blue light which represents the colour at the corresponding point of the image. The mixture of the red, green and blue components can therefore reproduce any of the colours of the original colour image, which can be any of the colours of the rainbow. A combined effect of the three image generators is therefore to reproduce a version of the colour image which is representative of the colour image formed within the field of view of the television camera.
Representing a colour image as red, green and blue signal samples provides a facility for transmitting, recording and reproducing the colour image in some way. However, in order to reduce an amount of information which must be transmitted in order to convey the colour image, known television transmission techniques and video image recording techniques convert the red, green and blue signals into colour difference signals, which are generally comprised of a luminance and a first and a second chrominance signal. The luminance signal is, for example, formed by combining the red, green and blue signal components of a pixel into a single component representative of the relative strength of the light in the image at the pixel location. The first of the chrominance signals is generated by forming a difference between the luminance signal and the red signal, and the second chrominance signal is formed from the difference between the luminance signal and the blue colour signal.
The colour difference signal format is one example of a signal format which forms a signal space in which the pixels of a colour video image can be represented, but which does not directly correspond with the red, green and blue components from which the colour video image was generated. As a result, not all values of the colour difference signal components representing a pixel within the colour difference space correspond to pixels within the signal space formed from the red, green and blue components of the colour image. For example, if the luminance component is at its minimum value of zero, then any non-zero value of the two chrominance signal components will result in a signal value which does not fall within the red, green and blue colour reference space. Similarly, if the luminance signal is at a maximum value which corresponds to white light, then any non-zero values of the two chrominance signals will also not fall within the red, green and blue reference space.
Any colour which does not fall within the red, green and blue reference space is an illegal colour. For the example of colour difference signals, any combination of the three components of the colour difference signals which results in a value which does not fall within the red, green and blue colour reference space will be an illegal value. Such illegal colour values can be produced when the colour images are transmitted or processed as, for example, colour difference signals. For example, video signals are often processed in this format to introduce video effects such as colour wash effects. As a result, values of the three colour reference space components can be produced which are illegal values within the red, green and blue reference space. If these illegal colour values are displayed within a colour image, colours can result which do not match with the legal parts of the image. The colour visual display unit reproducing the image may hard limit the colour value to a maximum value of the component which can be displayed, and the illegal pixels may be reproduced or processed in an unpredictable way.
In an article entitled xe2x80x9cLimiting of YUV Digital Video Signalsxe2x80x9d by V G Devereux from the Research Department, Engineering Division, of the British Broadcast Corporation dated December 1987, a method of converting illegal colour pixels in a form of YUV colour difference signals into legal colour pixels with respect to the red, green and blue (RGB) colour reference space is disclosed. This method changes the components of the pixels in the YUV colour difference space with respect to each other in order to convert the pixel in the corresponding red, green and blue colour reference space into a legal pixel.
Having regard to the above discussion, it will be appreciated that there is a general requirement to provide a method of processing colour video images in order to convert reliably illegal colour pixels of the images into legal colour pixels.
According to the present invention, there is provided a method of processing input signal samples representative of at least part of a colour video image to produce legalised colour signal samples representative of a legal colour version of the image, the method comprising the steps of generating adjustment factors from the input signal samples which when combined with the input signal samples have an effect of converting illegal colour pixels of the colour video image into legal colour pixels, softening the adjustment factors to reduce distortion produced in the legal colour version of the video image when combining the adjustment factors with the input signal samples and adapting at least one of said adjustment factors in dependence upon at least one other spatially associated adjustment factor which is associated with a position within the video image to the adjustment factor being adapted and combining the softened adjustment factors with the input signal samples to produce the legalised colour signal samples.
It has been discovered that after generating the adjustment factors and combining the adjustment factors with the input signal samples distortion may be introduced into the video image as a result of an effective bandwidth expansion of the video image caused by converting the illegal colour values to legal colour values. The term xe2x80x9csofteningxe2x80x9d is used to describe a process in which the adjustment factors are changed or adapted in some way so that when the adjustment factors are combined with the input signal samples this distortion is substantially reduced. The term xe2x80x9csoftxe2x80x9d or xe2x80x9csoftenxe2x80x9d therefore refers to adapting, changing or processing the adjustment factors in some way.
To provide the advantageous effect of reducing an amount of distortion in the video image, the step of softening the adjustment factors comprises a step of adapting at least one of said adjustment factors in dependence upon at least one other spatially associated adjustment factor which is associated with a position within the video image to the adjustment factor being adapted. In effect, however, the step of softening the adjustment factor may comprise the step of reducing the amplitude of the adjustment factor in dependence upon a relative amplitude of at least one spatially associated adjustment factor. To provide a facility for effecting the step of softening the adjustment factors more simply, the step of softening the adjustment factors may comprise the step of filtering the adjustment factors using a non-additive mixer. The non-additive mixer operates to soften the adjustment factors within a predetermined window of adjustment factors, by selecting the adjustment factor within the window which will have the most effect on the corresponding input signal samples; and setting the other adjustment factors to this value.
Since the step of softening the adjustment factors has an effect of changing the input signal samples in some way and therefore possibly introducing further distortion into the video image, advantageously the step of softening the adjustment factors may comprise the steps of determining whether each of the adjustment factors would have an effect when combined with a corresponding one of the input signal samples of changing the corresponding input signal sample and if the adjustment factor would have an effect of changing the corresponding input signal sample, softening the adjustment factor and otherwise setting the adjustment factor to a value in which the adjustment factor has no effect on the corresponding input signal sample.
In addition to the distortion produced in the video image as a result of applying the adjustment factors without softening, it has further been discovered that illegal colours can be produced as a result of distortion caused by aliasing errors. This distortion is produced by high frequency components of the video image in the analogue domain which are outside a maximum frequency which can be represented in accordance with a sampling rate of the signal samples. This produces distortion in the video image as a result of aliasing errors, which can cause legal pixels to become illegal. To provide an improvement by reducing the effect of this distortion and in order to provide a combined effect of softening the adjustment factors, the method of processing the input signal samples may further comprise the steps of generating an over-sampled version of the input signal samples by generating extra signal samples by interpolating between the input signal samples, the adjustment factors being generated for each of the input signal samples and the extra signal samples. By providing an over-sampled version of the input signal samples, the high-frequency out-of-band components in the video image which now appear within the in-band components of the over-sampled version of the input signal samples can be represented and therefore accommodated within the process of legalising the video image.
Advantageously, however, in combination with this over-sampling step, the step of softening the adjustment factors may comprise the step of selecting for each of the adjustment factors associated with an input signal sample and the associated at least one extra signal sample at least one of the associated adjustment factors which would have most effect on either the input signal sample or the associated extra input signal samples and correspondingly decimating the over-sampled version of the input signal sample before the step of combining the adjustment factors with the input signal sample.
Advantageously, however, in order to reduce a possibility of introducing distortion into the video image resulting from the adjustment factors, the step of selecting the adjustment factors may comprise the step of determining whether the associated adjustment factors would have an effect of changing at least one of the corresponding input signal sample and the at least one extra signal sample, and if the associated adjustment factors would not have an effect of changing at least one of the corresponding input signal samples and the at least one extra signal sample, selecting the adjustment factor associated with the input signal sample, and the step of correspondingly decimating the over-sampled version, comprises the step of selecting the corresponding input signal sample.
In order to ensure that all of the out-of-band components which are present in the input signal samples are represented within the over-sampled version of the input signal samples, the input signal may be over-sampled to the extent that the at least one extra signal sample may be a plurality of extra signal samples. In this case, therefore, after the softened adjustment factors have been combined with the input signal samples, the method may include the step of decimating the legalised colour signal samples. As would be understood from the foregoing explanation, the legal colour signal samples are those produced when the adjustment factors after softening are combined with the input signal samples and, in the case where the input signal samples have been over-sampled, the legalised colour signal samples will also be over-sampled after the over-sampled version of the input signal samples are combined with correspondingly over-sampled adjustment factors.
In order to further ensure that distortion and corruption are not introduced into the legalised colour signal samples which may cause the samples to once again become illegal, the method may include the step of filtering the legalised colour signal samples with an anti-aliasing filter having a bandwidth substantially equal to half the sampling frequency used to represent the legalised colour signal samples.
Having regard to the above explanation of effects which cause distortion to the video image, it has also been discovered that an effect of filtering the video image may be such as to cause legal colour pixels to become, once again illegal. To prevent aliasing errors when decimating it is conventional to filter the over sampled filter with an anti-aliasing low pass filter. It is this filtering which can cause legal colour pixels to become, once again illegal. As such, the method of processing the input signal samples is provided with a further improvement by incorporating into the step of filtering the over-sampled version of legalised colour signal samples the further steps of determining whether each of a plurality of the legalised colour signal samples and the extra legalised colour signal samples of the over-sampled version would change with respect to the corresponding input signal samples and the extra input signal samples when combined with the softened adjustment factors, and if at least one of the plurality of input and extra-legalised signal samples have changed with respect to the corresponding legalised and the corresponding extra-legalised colour signal samples, filtering the plurality of the legalised colour signal samples and the extra colour signal samples with the anti-aliasing filter, or if none of at least one of the extra-legalised colour signal samples and the associated extra-legalised colour signal samples have changed bypassing the step of filtering the plurality of the legalised colour signal samples and the extra colour signal samples.
Having regard to the effect over which the anti-aliasing filter has an influence over successive over-sampled versions of the legalised colour signal samples, which is known as the constraint length of the aliasing filter, the plurality of the legalised colour signals samples and the extra colour signal samples may correspond to the constraint length of the aliasing filter.
Distortion to the colour video image which may once again cause illegal colour values to be generated therein can also be produced when the over-sampled version of the legalised colour signal samples is decimated. For the this reason, the step of decimating the over-sampled version of the legalised colour signal samples comprises the steps of determining whether each of the legalised colour signal samples and the extra-legalised colour signal samples associated with each of the legalised colour signal samples were changed with respect to the corresponding input signal samples and the extra input signal samples when combined with the adjustment factors, and if at least one of said each input signal sample and said associated signal samples have changed with respect to said legalised colour signal samples performing the steps of filtering the legalised colour signal sample and the extra colour signal samples to form a corresponding decimated legalised colour signal sample, or if neither the each legalised colour signal sample nor the associated extra-legalised colour signal samples have changed, discarding the associated extra-legalised colour signal samples and forming the decimated signal sample from the legalised colour signal sample.
It has also been discovered that distortion can be caused to the video image after filtering and decimating the over-sampled version of the legalised colour signal samples. In order to provide an improvement by reducing further the possibility of causing distortion to the video image and thereby making otherwise legal colour signal samples into illegal colour signal samples, the method may further comprise the steps of generating further adjustment factors in dependence upon the decimated legalised colour signal samples and combining the further adjustment factors with the decimated legalised colour signal samples. To ensure that distortion is not introduced as a result of out-of-band harmonics being induced within the bandwidth of the colour video image, the method may advantageously include the step of softening the further adjustment factors and combining the softened further adjustment factors with the decimated legalised colour signal samples.
In order to provide further improvements in the generation of the adjustment factors and in particular to reduce a risk of adjustment factors changing input signal samples even though the input signal samples may correspond to legal colour values, the step of generating the adjustment factors may comprise the steps of comparing each of the input signal samples with a quantisation threshold and in dependence upon the comparison, setting the adjustment factor to a value at which the factor will have no effect when combined with the input signal sample.
A further advantage is provided to the step of generating the adjustment factors, by reducing a possibility of causing the legalised colour signal samples to become illegal during the step of combining the adjustment factors with the input signal samples, by including in the step of generating the adjustment factors may further comprise the step of pre-biasing the adjustment factors by scaling each of the adjustment factors with a biasing constant. A yet further advantage is provided to the step of generating the adjustment factors, by decreasing a possibility that when the adjustment factors are combined with the input signal samples, the legalised colour signal samples are more likely to produce legal colour pixels or, where appropriate, increasing the possibility of legal colour pixels being formed, the step of generating the adjustment factors may further comprise the step of pre-biasing the adjustment factors by scaling each of the adjustment factors with a biasing constant. By scaling the adjustment factors with a biasing constant, the effect of the adjustment factors is reduced or increased in dependence upon the value of the biasing constant. Effectively, therefore, those of the input signal samples which are determined to be representative of illegal colour pixels in the red, green and blue colour reference space will be moved further or to a lesser extent within the red, green and blue reference space respectively.
As explained above, although the input signal samples which are representative of the colour video image may have values with respect to a signal space which is different from the red, green and blue signal space, an example embodiment of the invention finds particular application where the input signal samples are colour difference signal samples having luminance and two colour difference signal components. As such, in the case where the adjustment factors are calculated and applied with reference to the red, green and blue colour reference space, the step of combining the adjustment factors with the input signal samples comprises the steps of converting the input colour difference signal samples into a colour reference signal samples having values with respect to three orthogonal colour reference axes of red, green and blue light, combining the colour reference signal samples with the adjustment factors and converting the combined colour reference signal samples into colour difference signal samples.
In the case where the input signal samples are legalised with reference to the red, green and blue colour reference axes, the step of converting the input colour difference signal samples into colour reference signal samples is performed before the adjustment factors are generated. The method may therefore comprise the step of converting the colour reference signal samples into a bipolar form, the adjustment factors being generated for the colour reference signal samples in the bipolar form. By converting the colour reference signal samples into signal samples with reference to a bipolar form, a further advantage is provided in improving the likelihood of legalising all colour values within the video image since the red, green and blue reference axes contain the minimum value of zero and, as such, are more likely to introduce a harsh adjustment of the red, green and blue values to the minimum value. Furthermore, by scaling the red, green and blue axes into a bipolar form, the legalised colour signal samples will be moved from illegal values towards the centre of the colour reference cube rather than to the edge of the colour reference cube. This further reduces the possibility of illegal colour values existing in the legalised video image.
Although the adjustment factors may be digital values which are added to the input signal samples in order to generate the legalised colour signal samples, in a preferred embodiment, the adjustment factors are scaling factors and the step of combining the adjustment factors with the input signal samples comprises the step of multiplying the adjustment factors with the input signal samples. As such, the adjustment factors may be represented on a scale between zero and one. In order to ensure that the effect of the adjustment factors on the input signal samples is greater when the input signal samples are representative of more illegal colour values than less illegal colour values, the step of softening the adjustment factors comprises the steps of reversing the scale of the adjustment factors from zero-to-one to one-to-zero and after the step of softening the adjustment factors, performing the step of reversing the scale of the adjustment factors from one-to-zero to zero-to-one. As a result of the reverse scaling of the adjustment factors, those adjustment factors which are greater and therefore have a greater effect on the input signal samples which would be values closer to zero are scaled to be closer to one, thereby increasing the effect of the softening process on these larger adjustment factors.
Accordingly in an aspect of the present invention, there is provided an image processing apparatus. Further features and aspects of the image processing apparatus are provided in the appended claims.